Point-neurons

• While historically biologically-inspired, they are very far from a biological neuron

• Point neurons are abstraction built to answer specific questions at a specific time

• From a ‘fit to neuroscience data’ point-of-vue, it’s a bad model BUT

• The point neuron has found its usefullness since it has become the default unit model in ANN

• Most of the complexity is abstracted away for better computational efficiency.

• But this neuron abstraction coupled with the way we train deep nets has shown cases where they fail miserably

  → continual learning and the catastrophic forgetting problem

Relevance of complexity (1)

• Multiple studies by many teams has point toward the importance of complexity both at the level of single neuron and in neural networks.

• Examples (non-exhaustive):

  • Payeur et al. (2021) : Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits

    when paired with short-term synaptic dynamics, regenerative activity in the apical dendrites, and synaptic plasticity in feedback pathways, a burst-dependent learning rule can solve challenging tasks that require deep network architectures. […]well-known properties of dendrites, synapses, and synaptic plasticity are sufficient to enable sophisticated learning in hierarchical circuits.

  • Perez-Nieves et al. (2021): Neural heterogeneity promotes robust learning

    the performance of spiking neural networks trained to carry out tasks of real-world difficulty, with varying degrees of heterogeneity, and found that it substantially improved task performance.

Relevance of complexity (2)

• Zeldenrust, Gutkin, and Denéve (2021): Efficient and robust coding in heterogeneous recurrent networks

  networks consisting of a heterogeneous population of different neuron types are both more efficient and more robust against correlated noise.

• Jones and Kording (2021): Might a Single Neuron Solve Interesting Machine Learning Problems Through Successive Computations on Its Dendritic Tree?

  simple model where the dendrite is implemented as a sequence of thresholded linear units. We manipulate the architecture of this model to investigate the impacts of binary branching constraints and repetition of synaptic inputs on neural computation. […] models with such manipulations can perform well on machine learning tasks, such as Fashion MNIST or Extended MNIST. We find that model performance on these tasks is limited by binary tree branching and dendritic asymmetry and is improved by the repetition of synaptic inputs to different dendritic branches.

What are dendrites ?

• Dendrites are major elements of biological neurons

• They are projections from the cell’s soma into the surrounding environment

• They make up most of the substrate of connections between cells (there can also be sub-structures called spines or not)

• Dendrites have extremely diverse morphologies

(Donohue and Ascoli 2008)

• In biology structure and function are typically linked => there should be a dendritic function explaning this diversity

Conceptual usefullness of dendrites

• Beniaguev, Segev, and London (2021) showed that a layer 5 pyramidal neuron I/O could be approximated using a 7 layers ANN composed of point neurons.

(Beniaguev, Segev, and London 2021)

• However, no matter how deep an ANN is, it will always be subject to catastrophic forgetting.

• Hypothesis : Dendritic units can help minimize the catastrophic forgetting problem in continual learning while still maintaining a low compute overhead

Dendrite properties (1) : coincidence detectors and information routing

• Dendrite can act as coincidence detectors.

• Differing information content is routed to different part of the dendritic arbor

• Leading to the potential emergence of functional subdomains

Dendrite properties (2) : Linear and non-linear computations

• Neurons contains active or passive dendrites.

• Passive dendrites show only linear or sublinear input summation.

• Active dendrite can show supralinear outpout.

• Same dendritic branch stimulation can results in linear or supra-linear output depending on stimulation conditions.

Dendrite properties (2) : Linear and non-linear computations

• Same dendrite stimulation can result in non-linearity whereas that is not the case for between dendrites stimulations

(London and Häusser 2005)

Dendrite properties (3) : Dendritic plateau potentials

• Dendrites can integrate information over long time periods

(Antic et al. 2010)

Dendrite properties (4) : Dendrite hetereogeneity

• Dendrites functional properties depends on their morphology hence lots of room for variability / hetereogeneity in dendrites properties.

(Tran-Van-Minh et al. 2015)

Dendrites relevance to neuronal computations

Case study 1: Tzilivaki, Kastellakis, and Poirazi (2019) - Challenging the point neuron dogma: FS basket cells as 2-stage nonlinear integrators

• Non linear dendrites in interneurons help store memories using fewer ressources
• Linked memory are stored in common dendrites, through synapse clustering

Dendrites relevance to neuronal computations

Case study 2: Jones and Kording (2021) - Might a Single Neuron Solve Interesting Machine Learning Problems Through Successive Computations on Its Dendritic Tree?

• Modelling dendrites as a sequence of thresholded linear unit solves typical ML benchmarks (non-CL scenarios)

• Benefit from having the same input onto several branches of the dendritic tree = importance of synaptic input repetition

• Performances on-par with a fully-connected neural network (FCNN)

Dendrites relevance to neuronal computations

Case study 3: Grewal et al. (2021) - Going Beyond the Point Neuron: Active Dendrites and Sparse Representations for Continual Learning and Iyer et al. (2022) - Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments

• Adding dendrite abstractions in an MLP help preventing catastrophic forgetting both in abstract task (permutedMNIST) and reinforcement learning (RL) tasks.

• Context-driven task-specific subnetworks activation gated by dendritic inputs

Dendrites relevance to neuronal computations

Case study 3: Grewal et al. (2021) - Going Beyond the Point Neuron: Active Dendrites and Sparse Representations for Continual Learning and Iyer et al. 2022 - Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments

• Dendrite specialize in few tasks.

• Contradict the hypothesis that a dendritic neuron is equivalent to a multilayer network as classical deep networks are incapable of performing well due to catastrophic forgetting, regardless of network depth

Conclusion

• Dendrites’ computational role is far from solved

• Neuroscience informs models design which in turn inform back neuroscience. That’s a neat loop

• Depending upon the task, adding dendrites could increase computational overhead without benefits i.e. in non continual learning scenario there might be few, if any, benefits to adding dendrites

• In continual learning settings, dendrites could be a powerfull mechanism to design networks more robust to catastrophic forgetting

• There is still a lot to open ended questions here.

  • Is the computational overhead of dendrite worth the cost ?
  • Which are the scenarios where having dendrites is helping most to perform the task ?
  • What are the dendritic mechanisms that would enhance classical deep nets the most ?